Skip to content

Packages upgrade - #2535

Merged
Didayolo merged 6 commits into
developfrom
packages_upgrades
Sep 15, 2026
Merged

Didayolo merged 6 commits into
developfrom
packages_upgrades

Conversation

@ObadaS

@ObadaS ObadaS commented Sep 11, 2026 •

Copy link
Copy Markdown
Collaborator

A brief description of the purpose of the changes contained in this PR.

  • Updated djangorestframework in pyproject and some other packages via uv lock --upgrade
  • Updated Gunicorn and Uvicorn to latest
  • Updated boto3 to latest (and removed s3transfer and botocore from pyproject, they are dependencies of boto3 and make upgrading boto3 more complicated for no reason)
  • Upgraded DOMPurify, marked and katex packages
  • Upgrading django-oauth-toolkit requires a migration. The migration is run automatically in the docker-compose.yml file, but make sure to backup your database before.

I think this fixes all the points from #2502

Checklist

  • Code review by me
  • Hand tested by me
  • I'm proud of my work
  • Code review by reviewer
  • Hand tested by reviewer
  • CircleCi tests are passing
  • Ready to merge

@Didayolo

Didayolo commented Sep 15, 2026 •

Copy link
Copy Markdown
Member

Trying to test this but it looks like I'm polluted by the recent #2499, #2533:

Running migrations:
  Applying competitions.0063_remove_competition_allow_robot_submissions...Traceback (most recent call last):
  File "/.venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
           ~~~~~~~~~~~~~~~~~~~^^^^^
psycopg2.errors.UndefinedColumn: column "allow_robot_submissions" of relation "competitions_competition" does not exist


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/./manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
    ~~~~~~~~~~~~~~~^^
  File "/.venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv
    self.execute(*args, **cmd_options)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute
    output = self.handle(*args, **options)
  File "/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 111, in wrapper
    res = handle_func(*args, **kwargs)
  File "/.venv/lib/python3.13/site-packages/django/core/management/commands/migrate.py", line 353, in handle
    post_migrate_state = executor.migrate(
        targets,
    ...<3 lines>...
        fake_initial=fake_initial,
    )
  File "/.venv/lib/python3.13/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
        state, plan, full_plan, fake=fake, fake_initial=fake_initial
    )
  File "/.venv/lib/python3.13/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
        state, migration, fake=fake, fake_initial=fake_initial
    )
  File "/.venv/lib/python3.13/site-packages/django/db/migrations/executor.py", line 255, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/.venv/lib/python3.13/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.app_label, schema_editor, old_state, project_state
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/.venv/lib/python3.13/site-packages/django/db/migrations/operations/fields.py", line 174, in database_forwards
    schema_editor.remove_field(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        from_model, from_model._meta.get_field(self.name)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/.venv/lib/python3.13/site-packages/django/db/backends/base/schema.py", line 829, in remove_field
    self.execute(sql)
    ~~~~~~~~~~~~^^^^^
  File "/.venv/lib/python3.13/site-packages/django/db/backends/postgresql/schema.py", line 48, in execute
    return super().execute(sql, None)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/django/db/backends/base/schema.py", line 204, in execute
    cursor.execute(sql, params)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sql, params, many=False, executor=self._execute
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/.venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/.venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/.venv/lib/python3.13/site-packages/django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
           ~~~~~~~~~~~~~~~~~~~^^^^^
django.db.utils.ProgrammingError: column "allow_robot_submissions" of relation "competitions_competition" does not exist

Putting it here to try to debug.

@Didayolo

Copy link
Copy Markdown
Member

I think it is only my local instance that has problem. Going to fix it.

@ObadaS

ObadaS commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

This does look like missing migrations or something, not related to this PR

@Didayolo

Didayolo commented Sep 15, 2026 •

Copy link
Copy Markdown
Member

This does look like missing migrations or something, not related to this PR

Yeah it's because I fixed migrations with #2533 but my local history is unhealthy. I'll just re-install my instance from scratch.

Testing the develop upgrade is important though.

@Didayolo
Didayolo marked this pull request as ready for review September 15, 2026 14:23
@Didayolo
Didayolo merged commit bc930d6 into develop Sep 15, 2026
1 check passed
@Didayolo
Didayolo deleted the packages_upgrades branch September 15, 2026 14:23
Didayolo added a commit that referenced this pull request Sep 18, 2026
* Update version.json

* add option to forbid the compute worker from pulling the competition image automatically

* rename feature name

* packages bump via uv lock --upgrade; bump django 5.2.15 -> 5.2.16 (patch)

* rebase and update some packages via uv lock --upgrade

* various packages upgrades with uv lock --upgrade

* Add pagination to GET /api/competitions and fix participating_in leak

Paginate the competition list endpoint (LargePagination), and update
the Organizing/Participating tabs in competition_list.tag to
consume the new {count, next, previous, results} shape with
next/previous buttons (hidden on a single page). Fix the navbar
competition search to read response.results instead of assuming a
bare array.

Fix participating_in filter incorrectly including competitions the
user organizes: creators/collaborators are auto-added as approved
participants (Competition.save()), so they leaked into the
Participating tab. Applied the same fix to the public competitions
endpoint's participating_in filter and updated its tests. Added
coverage for the list endpoint's pagination and participating_in
behavior. Minor renames for clarity (change_page, tab labels).

* remove ram limit for site_worker

* update limit to 15 GB, same as default quota

* Update docker-compose.yml

* queue creation error improved

* allow users to add collaborators to queues when the queue adding limit is reached. Fixed user lookup to not show deleted users in the collaborators dropdown

* Remove robot submissions and bot user feature

Remove the is_bot user flag and allow_robot_submissions competition setting, along with the permission bypasses they granted (unlimited submissions, auto-approval as participant, and unrestricted re-run/secret-check bypass on any submission). Also removes the related UI, docs, and tests.

* Fix migrations

* privacy link updated

* Fix organization name not showing on leaderboard for multi-task competitions

Propagate organization when creating chiled submissions during multi-task split, phase migration, and re-run so that leaderboard entries for organization submissions on multi-task competitions show the organization name instead of falling back to the submitter's username.

* Merge pull request #2487 from codalab/throttling

API Throttling

* restrict get_details to submission owner or admin

Anonymous and non-owner requests could previously view submission
details when hide_output was off. Now only the owner or an admin
can access them. Adds tests covering the fix.

* Switch to quay.io for mc and minio images (#2537)

* switch to quay.io for mc and minio images

* Update documentation

---------

Co-authored-by: Obada Haddad <obada.haddad@lisn.fr>
Co-authored-by: didayolo <adrien.pavao@gmail.com>

* Deny anonymous access to /api/submissions/ list and retrieve (#2536)

* Deny anonymous access to /api/submissions/ list and retrieve

Anonymous GETs returned full submission records (filenames, status
details, fact sheet answers) for any finished leaderboard submission,
ignoring is_public. Now returns an empty queryset instead; public
leaderboard data is already served by get_leaderboard in PhaseViewSet

* Add missing raise

* Fix test and remove debug prints

---------

Co-authored-by: didayolo <adrien.pavao@gmail.com>

* Packages upgrade (#2535)

* update djangorestframework in pyproject and packages in uv.lock with uv lock --upgrade

* update boto3 (removed s3transfer and botocore, they are dependencies), update gunicorn and uvicorn to latest

* update packages in pyproject

* update django-oauth-toolkit and related packages

* update dompurify

* add DOMPurify to marked

---------

Co-authored-by: Obada Haddad <obada.haddad@lisn.fr>

* Submission API: Enforce phase start/end window on submission creation (#2496)

* Enforce phase start/end window on submission creation

Phase.is_active returned True unconditionally when end was unset,
never checking start — a not-yet-started phase with no end date was
treated as active. SubmissionCreationSerializer.validate() also never
checked is_active or can_user_make_submissions(), so submissions could
be created via the API before a phase started or after it ended.

- Fix Phase.is_active to check start regardless of whether end is set
- Reject submission creation when the target phase is not active or
  the user has hit their submission limit, returning a 400 error
- Add tests for both new fixes

* Fix flake8

---------

Co-authored-by: didayolo <adrien.pavao@gmail.com>

* compute worker workaround for api throttling

* Add External Competitions feature (#2508)

* Add External Competitions feature: aggregate competitions from other Codabench/CodaLab instances

New app — external_competitions
- Models: ExternalPlatform (a registered remote instance), ExternalCompetition (a fetched competition), ExternalFetchLog (per-sync audit trail with SUCCESS/PARTIAL_SUCCESS/FAILURE status, counts, and error messages)
- admin.py registrations for all three models with list filters/search
- Initial migration

Fetchers
- fetchers/codabench_fetcher.py — paginated fetch from a Codabench instance's public competitions API, with a page-fetch throttle and a MAX_PAGES safety cap
- fetchers/codalab_fetcher.py — flat-list fetch from a CodaLab instance
- fetchers/exceptions.py — PartialFetchError, letting a fetcher hand back whatever it collected before a later page failed

Sync task
- fetch_sync.py — fetch_external_competitions Celery task (feature-flag gated, scheduled daily via CELERY_BEAT_SCHEDULE) and sync_platform, which diffs fetched data against the DB (create/update/delete), skipping the delete step on a partial fetch to avoid dropping valid competitions from unfetched pages

API
- api/serializers/external_competitions.py — ExternalCompetitionSerializer, ExternalPlatformFilterSerializer
- api/views/external_competitions.py — list view with search/platform filtering, and an unpaginated platforms list for the filter UI
- api/urls.py — both routes registered only when EXTERNAL_COMPETITIONS_ENABLED is on

Frontend
- external_competitions/urls.py + views.py + templates/external_competitions/public.html — the public page route
- static/riot/external_competitions/external_competition_list.tag — the competition list/filter/pagination UI
- client.js — API helpers for the two new endpoints
- public-list.tag — banner linking to the External Competitions page (shown only when the feature is enabled)
- base.html / context_processors.py — expose the feature flag and public URL to the frontend

Settings
- EXTERNAL_COMPETITIONS_ENABLED flag (settings/base.py, .env_sample), off by default
- external_competitions added to INSTALLED_APPS

Test data
- factories.py — ExternalPlatformFactory, ExternalCompetitionFactory

Tests
- external_competitions/tests/test_fetchers.py — unit tests for both fetchers (pagination, throttling, MAX_PAGES, error propagation, partial-fetch handling)
- external_competitions/tests/test_fetch_sync.py — unit tests for sync_platform/fetch_external_competitions (create/update/delete diffing, partial-success handling, per-platform failure isolation)
- api/tests/test_external_competitions.py — API tests for both endpoints, plus flag-gated URL registration tests

Documentation
- New External-Competitions.md page: enabling the flag, admin fields, sync mechanics, and registration/unregistration instructions for other platform maintainers
- zensical.toml nav entry
- Tip added to the deploy guide pointing self-hosters at registering their instance

* flake-8 fixes

* registration instructions added to external competitions page

* renamed fetch_sync.py to tasks.py so that celery can run the task periodically, updated codabench fetcher to not use the next url from the response but instead use a page counter for next page, related updates to docs and tests

* ui updates to differentiate external page from public page.

* Some small fixes

* Fix spinner

* Avoid duplicated CSS

---------

Co-authored-by: didayolo <adrien.pavao@gmail.com>

* Option to stop compute worker from sending logs and/or prediction files to Codabench (#2460)

* add option in compute worker env to not send logs to the instance, instead writing them in a local file

* rename the No Cleanup env variable, add documentation

* use real boolean values

* update logs_loguru to inclue new tasks variable names to color them

* change variable name to use boolean

* fix some syntax

* add option to forbid the compute worker from sending predictions to the codabench instance storage

* add better coloration for some logs in the compute worker

* update documentation

* add error when copying failes when in no prediction upload mode

---------

Co-authored-by: Obada Haddad <obada.haddad@lisn.fr>

* Revert "Compute worker workaround for API throttling"

* Revert "Merge pull request #2487 from codalab/throttling"

This reverts commit 17416df.

---------

Co-authored-by: Obada Haddad <obada.haddad@lisn.fr>
Co-authored-by: wlln <105347618+wlln@users.noreply.github.com>
Co-authored-by: Adrien Pavão <adrien.pavao@gmail.com>
Co-authored-by: Ihsan Ullah <ihsan2131@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stored XSS in competition pages / terms / phase descriptions

2 participants